Automatic Metadata Annotation of Images via a Two-Level Learning Framework

A key problem for handling multimedia data in the semantic web is finding a way to associate concepts from ontologies to multimedia data items at an acceptable cost. This paper describes experiments with a system to assign automatically keyword metadata descriptors to unlabelled images. Learning to automatically match low level image features, like colour or texture to high level concepts (the so called semantic-gap problem) is very challenging. The usual approach is to design a learning machine or classifier to learn low-level feature vectors for high-level concept classification as a single-step (direct) mapping function. These systems often do not perform well for large numbers of classes. We present a two-level supervised learning framework for effective image annotation. In the first level induction stage, colour and texture feature vectors are classified individually into their corresponding outputs, i.e. colour and texture terms. Then, the colour and texture terms as middlelevel features are classified into the target high-level conceptual classes during the second level induction stage. Three experimental studies are described in this paper. Experimental results using vocabularies of 60 and 150 keywords are reported, based on single step Support Vector Machines, two step Support Vector Machines, and k Nearest Neighbour. In the final experiment a comparison between human and automatic metadata annotation is described. Results show promise that the techniques will scale and perform acceptably for practical retrieval.